Skip to content

fix(swebench): keep tags on the base commit's ancestors in the anti-cheat git scrub - #6372

Open
Rogerffff wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
Rogerffff:fix/swebench-keep-ancestor-tags
Open

Rogerffff wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
Rogerffff:fix/swebench-keep-ancestor-tags

Conversation

@Rogerffff

Copy link
Copy Markdown

What does this PR do?

Closes #5745 by implementing the proposal there.

anti_cheat_setup.sh deletes every tag, including the tags on ancestors of the task's base commit. Projects that compute their version from git describe at runtime then report 0+untagged.... In SWE-Gym's pandas 2.1 tasks, pandas/conftest.py then fails to import, so the agent cannot run any test (#5745: 113 of 114 tested 2.1 tasks, plus 21 tasks on 2.0 and 1.5 where some tests fail). Grading is not affected, because /verify runs in an unscrubbed sandbox.

  • anti_cheat_setup.sh: record the tags on ancestors of HEAD (git --no-replace-objects for-each-ref --merged=<base> refs/tags) and recreate them after the refs are deleted, before git gc. Everything else is still removed: later and side-branch tags, branches, remotes, reflogs and unreachable objects. --no-replace-objects keeps a replace ref from passing a later commit off as an ancestor. The GIT_CLEANUP line gains tags_kept=N.
  • New tests/test_anti_cheat_setup_script.py (10 tests) runs the real script on throwaway repositories, with HOME and GIT_CONFIG_GLOBAL in tmp_path.
  • Four docs that said the scrub drops all tags now say ancestor tags stay.

On the questions in #5745: I found no recorded reason for deleting ancestor tags (#2579, #3327). They point only at commits the agent can already see, though annotated tags keep their message. The change is in the shared script, so all 14 servers that run it get it (not 12, as #5745 said); I checked images from five of them (see Validation).

Validation

gym env test +entrypoint=resources_servers/swebench +should_validate_data=true
gym env test +entrypoint=resources_servers/swe_gym +should_validate_data=true
gym env test +entrypoint=resources_servers/r2e_gym +should_validate_data=true
pre-commit run --all-files

On Linux (Debian 12 x86_64, git 2.39.5, bash 5.2, Python 3.13.14): swebench 177, swe_gym 105 and r2e_gym 64 passed, data validation passes, and pre-commit passes (guardwords skipped without the private catalog). The new tests on unpatched main: 8 failed, 2 passed.

Image controls on Linux: fresh containers per published image (no scrub, current script, this PR; scrubbed as apply_anti_cheat_setup does), then the task's test command at the base, with no patch:

Server Task none / current / this PR
swe_gym conan-io__conan-13610, iterative__dvc-1651, pydantic__pydantic-8500 the same tests pass in all three (2, 1, 44)
swe_gym pandas-dev__pandas-53958 11 passed / conftest import error, 0 tests / 11 passed
r2e_gym pandas e0c5b878 12 passed / conftest import error, 0 tests / 12 passed
r2e_gym numpy f9bed20b 1321 passed, 51 failed / test_numeric.py fails to collect (int('0+untagged') in hypothesis) / 1321 passed, 51 failed
swe_rebench scalameta-3707 sbt "show version" works / sbt cannot load build.sbt (git describe exits 128) / works
swebench matplotlib-24149, xarray-3151 the same tests pass in all three (767, 66); only the version string changes
swe_rebench, scale_swe pandas-58970, geopandas pr2415 the same in all three: the image has no tags, or the task's pre_commands remove them first

Nothing after the base stays reachable with either script. The swe_gym rows and the rollouts ran the previous revision of this branch, without --no-replace-objects, which changes nothing on those images (they have no replace refs); the other rows ran this revision.

Rollouts

One smoke rollout per side on pandas-dev__pandas-53958: deepseek-flash through inference_provider, OpenCode through single_agent_turn, Docker sandbox, one sample each.

main this PR
Tags in the agent's sandbox none 92, all ancestors
Agent's own pytest stops at conftest import (pandas 0+untagged... is installed) 11 passed
/verify resolved, 1.0 resolved, 1.0

On main the agent read the conftest failure as an environment problem ("pytest was blocked by an unrelated pyarrow/pandas version warning in this environment", in its final message) and checked its change with hand-written python -c snippets instead. The equal rewards say nothing about this change: both agents copied the fix from the newer pandas in the image's conda package cache (#5744).

Command
CFG=(--config resources_servers/swe_gym/configs/swe_gym_opencode.yaml
     --config nemo_gym/sandbox/providers/docker/configs/docker.yaml
     --model-type inference_provider)
OVR=(++sandbox.docker.create.network=host
     ++swe_gym_opencode_agent.responses_api_agents.opencode_agent.token_id_capture=false
     ++swe_gym_opencode_agent.responses_api_agents.opencode_agent.timeout=2400
     ++swe_gym_opencode_agent.responses_api_agents.opencode_agent.context_window=131072)
gym env start "${CFG[@]}" "${OVR[@]}"
gym eval run --no-serve "${CFG[@]}" "${OVR[@]}" --input pandas_53958_task.jsonl --output rollouts.jsonl --limit 1

network=host lets OpenCode in the sandbox reach the model server on 127.0.0.1. On main the agent ran a wrong node ID (TestPDApi::test_api_typing), but pytest failed earlier, at conftest import.

Compatibility and benchmark impact

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

Proposed labels (I cannot apply labels): bug, area:environment, needs-review.

…heat git scrub

anti_cheat_setup.sh deleted every tag, including the tags on ancestors
of the task's base commit. Projects that take their version from git
describe then report 0+untagged..., which in SWE-Gym's pandas 2.1 tasks
breaks pandas/conftest.py and stops the agent from running tests in its
sandbox.

Record the tags merged into the base before deleting refs, and recreate
them before git gc. Ancestry is computed with --no-replace-objects,
since replace refs are deleted too and could otherwise make a later
commit look like an ancestor. Tags that are not on an ancestor, other
branches, remotes and reflogs are still removed. Add a regression test
that runs the script on throwaway repositories, and update the four docs
that said the scrub drops all tags.

Closes NVIDIA-NeMo#5745

Signed-off-by: Rogerffff <143265077+Rogerffff@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Nov 18, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added risk:mid Autobot PR risk: mid bug Something isn't working area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers needs-review PR is ready for code review and waiting on a reviewer labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working community-request Issue reported or requested by someone from the community needs-review PR is ready for code review and waiting on a reviewer risk:mid Autobot PR risk: mid

Projects

None yet

2 participants